feat: update UnixFS/MFS with auto sharding #8114
Conversation
schomatis
force-pushed
the
feat/deps/update-unixfs-mfs-auto-sharding
branch
from
01639e4
to
811ea25
Compare
js counterpart to ipfs/kubo#8114 Changes the `shardSplitThreshold` parameter to mean the size of the final DAGNode (including link names, sizes, etc) instead of the number of entries in the directory. Fixes: #149 BREAKING CHANGE: `shardSplitThreshold` now refers to node size, not number of entries
| // TEMP: setting global sharding switch here | ||
| uio.UseHAMTSharding = cfg.Experimental.ShardingEnabled | ||
| uio.HAMTShardingSize = int(256 * units.KiB) |
There was a problem hiding this comment.
Seems like this should live in go-unixfs since we'd like people to be using the automatic sharding unless they opt not to
There was a problem hiding this comment.
As it stands they will be using automatic sharding by default for go-ipfs, not for go-unixfs as a standalone library (independent of BitSwap, which is the one who introduce this restriction in the first place).
Do we also want this to be a default for go-unixfs as a library? (In that case we would set the HAMTShardingSize default value to 256KiB in unixfs and not here.)
There was a problem hiding this comment.
Do we also want this to be a default for go-unixfs as a library? (In that case we would set the HAMTShardingSize default value to 256KiB in unixfs and not here.)
Yes, I think we should just put this in go-unixfs. If at some point that library starts taking sharding sizes in a non-global fashion we can deal with it there. The point of defaults is to give people something sane to start with if they're not deeply "in the know" so we might as well give it to go-unixfs consumers as well.
There was a problem hiding this comment.
Moving to UnixFS.
There was a problem hiding this comment.
Added in ipfs/go-unixfs@83ad983.
Will remove this after we update UnixFS/MFS dependencies with latest fixes.
aschmahmann
force-pushed
the
feat/deps/update-unixfs-mfs-auto-sharding
branch
from
5983810
to
a369655
Compare
schomatis
force-pushed
the
feat/deps/update-unixfs-mfs-auto-sharding
branch
from
19a8f06
to
3a845d1
Compare
|
Sharness are passing which is encouraging. Rebasing on master. |
schomatis
force-pushed
the
feat/deps/update-unixfs-mfs-auto-sharding
branch
from
0dd3b6b
to
56866cf
Compare
|
Closed by #8547 |
js counterpart to ipfs/kubo#8114 Changes the `shardSplitThreshold` parameter to mean the size of the final DAGNode (including link names, sizes, etc) instead of the number of entries in the directory. Fixes: #149 BREAKING CHANGE: the `shardSplitThreshold` option has changed to `shardSplitThresholdBytes` and reflects a DAGNode size where sharding might kick in
TODO:
cfg.Experimental.ShardingEnabled. Optionally maybe also add a new experimental option with the sharding threshold value if we want to move away from the 256 KiB hardcoded now.Internal.UnixFSShardingSizeThresholdUnixFS PR (already merged): feat: switch to HAMT based on size go-unixfs#91
MFS PR: support threshold based automatic sharding and unsharding of directories go-mfs#88
Closes #8106.
Closes #8148.